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Suntry:  A  new  iterative  procedure  for 
rinding  the  solutions  of  genes  is  presented. 

/ 

ON  A  NEW  ITERATIVE  ALGORITHM  FOR  FINDING  THE  SOLUTIONS 
OP  GAMES  AND  LINEAR  PROGRAMtlNG  PROBLEMS 

Richard  Bellman 

§1 .  Introduction. 

The  purpose  of  this  paper  is  to  present  a  new  iterative  pro¬ 
cedure  for  finding  the  solutions  of  games.  Since,  as  is  now  well 
known,  every  linear  progranming  problem  of  conventional  type  can 
be  transformed  into  a  symmetric  game,  this  technique  may  be  applied 
to  the  solution  of  linear  programing  problems.  It  was  actually 
with  this  purpose  in  mind*  that  the  method  was  developed.  Our  aim 
is  to  obtain  a  procedure  which  converges  more  rapidly  than  either 
the  statistical  method  of  Brown,  or  the  differential  equation 
approach  of  Brown  and  von  Neumann. 

We  shall  first  present  a  variant  of  the  differential  equation 
approach  which  converges  more  rapidly  than  the  original.  Carrying 
this  approach  to  its  logical  limit  we-obtain  a  process  with  an 
exponential  rate  of  convergence.  The  discrete  analogue,  obtained 
by  replacing  the  differential  equation  by  a  difference  eqaation, 
furnishes  the  new  iterative  algorithm. 

In  replacing  the  differential  equation  by  a  difference  equa¬ 
tion,  the  expression  dx/dt  is  replaced  by  (x(t4li)-x(t) )/h  with  t 
taking  the  values  0,  h,  2h,  etc.  At  the  N^^  step  the  error  will 
be  proportional  to  e  .  Since,  in  general,  h  must  be  taken  small 
to  Insure  that  the  solution  of  the  difference  equation  is  close  to 
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the  lolution  of  the  differential  equation,  a  large  number  of 
Iterations  will  be  required  to  obtain  an  accurate  approximation. 

To  counteract  this  unpleasant  feature  we  have  devised  a  com- 
blnatlon  of  a  continuous  and  discrete  Iteration  process  which 
seems  well  suited  to  hand  computation.  For  games  of  fairly  large 
slse  It  would  seem  that  the  most  economic  policy,  taking  account 
of  the  cost  of  both  hand  and  machine  computation,  would  be  to  use 
the  continuous  Iteration  technique  up  to  a  certain  point  and  then 
let  the  machines  take  over  using  a  discrete  technique. 

The  method  has  been  tried  on  a  number  of  examples,  and  In  all 
cases  It  seems  bo  yield  the  value  of  the  game  very  rapidly.  It  Is 
conceivable  that  In  certain  linear  programming  problems  It  might 
be  desired  only  to  determine  the  value  of  the  maximum  or  minimum 
rather  than  the  variables  which  yield  the  critical  value.  For 
problems  of  this  type  the  method  seems  admirably  suited. 

I  would  like  to  express  my  sincere  appreciation  to  Bernice  Brown, 
without  whose  enthusiastic  Interest  and  cooperation  the  method  would 
have  perished  stlll-bom. 


Let  us  consider  a  symmetric  game  characterized  by  the  siMf u'>  - 
symmetric  matrix  and  set 


N 


(•>)  ''1  ■  ^ 


(e)  <>(u^)  -  Max  15,u^3 


(d)  M*)  -  ^  Mui) 


(1) 


The  Bysten  of  differential  equationa  to  be  used  to  yield  a 


solution  is 

(2)  "■  ^<*)*i* 


*1(0) 

with  0^  ^  0, 


1. 


\ 


An  ingenious  argument  from  this  point  on  shows  that  — >  0 

as  t  — >00,  which  means  that  the  x^(t)  must  have  cluster  points 
which  furnish  solutions  to  the  game.  These  may  be  obtained  by 
inspection  of  the  numerical  results. 

The  convergence  of  Muj^)  is  of  the  order  of  l/t  as  t  —^00. 


§5.  A  Variant  of  the  Brown -von  Neumann  Technique. 

In  an  effort  to  speed  up  convergence,  let  us  see  what  happens 
if  we  replace  (2)  of  §2  by 


(1) 

where 

N 

(2)  <>^(x)  -  . 

We  need  no  longer  have  uniqueness  of  solution  if  0  <Q«  1.  This 
however,  is  no  handicap  since  the  method  will  show  that  any  solution 
of  (1)  will  have  the  property  that  ^(u^)  — >  0. 
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Rtpeatlng  the  Bx*oim>von  Neumann  arsument  we  have.  If  >  0, 

(3)  ^ 'ij  "  *«<*^*j  J 

/ 


Htnea  for  all  t, 

(“)  *<"1)  •  ^ 

-U*) 

« 

Sumlng  over  1  thle  yields 

■Inoe 


by  virtue  of  the  skew  synmetry  a^j  ■  ~  a^^. 
Let  us  now  use  the  fact  that 

(7)  4>^(x)  -  X  *(“i)  ik(«<)  ( 


Where  k(Q^  is  an  appropriate  constant.  Hence 


(8) 


^  o(?r 


Writing  V  «  (8)  beooaes 


(9) 


Integrating  this  Inequality  we  obtain 


^  dc.t  ;k;)Tiv^ 

whex*e  ki  and  kt  are  positive  constants. 

The  Interesting  feature  about  this  result  la  that  the  smaller  oC 
Is  chosen y  the  more  rapid  the  convergence. 

• 

§4.  The  Limiting  Caeeo(«  0. 

The  above  observation  leads  us  to  consider  the  case  0.  The 
differential  equation  takes  the  form 

(1)  “3^  “  —  i}(x)x^, 

where 

f(u^)  -  1  If  Uj^  >  0, 

■  0  If  U^  ^  Oy 

<r(x)  -  ’■ 


(2) 


/ 
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We  have 


(3) 


du.  "  d*  “ 


TX 


N 


and,  as  above, 


(*») 


N  du  " 

^  -  ><*)  ^  VK 


Hence 


N 


N 


(9) 


■4e  if(Ul))  — ♦(«)  ^  U^f(u^), 


which  yields 
N 


(10) 


Sr 


)  -  aie 


V 


z)dti 


Since  fF(z)  ^  1  as  long  as  the  systen  Is  not  In  Its  equilibrium 
position  we  see  that 

N 

(11)  ^ ^ 

as  long  as  one  f  is  positive.  This  shows  that  the  convergence  Is 
exponential. 


§5-  The  Differtne#  Equation. 

For  computing  purposes  we  write  (l)  Of  la  the  foxwi 

(1)  -  x^(n)  -  h  |f(u^(n))  -  t(x(n) )s^(n)  J  , 


or 


(2)  x^(n-fl)  -  x^(n)(l-h>(x(n))  ♦  hf(ui(n)). 

In  this  fonDf  although.^  x^(n)  Is  equal  to  1  for  all  n»  an  Indi¬ 
vidual  x^  may  go  negative  to  a  small  degree. 

To  avoid  this*  we  replace  (l)  by 

(3)  *i(n+l)  ^  x^(n)  -  hj^f(uj,(n))  -  t(x(n))x^(n-i'l)  J  , 

- 

which  yields 


(4)  Xj^(n+1) 


x^(n)  +  hf(uj^(n)) 
1  h<r(x(n)) 


It  Is  probably  true  that  Max  |T>»u^(nQ  goes  to  sero  as  n  — >  oo , 
and  exponentially  fast,  although  we  have  not  proved  this.  Compu¬ 
tation  on  particular  games  with  small  values  of  h  would  seem  to  bear 
this  out.  A  discussion  of  some  of  these  will  be  given  subsequently. 

§6.  A  Procedure  for  Speeding  up  the  Discrete  Iteration. 

Since  f(uj^(n)),  and  hence  ^(x(n)),  depend  only  upon  the  sign  of 
u^(n),  not  its  magnitude,  it  Is  clear  that  the  Iteration  formula  id  11 
'  be  the  same  for  long  stretches.  Furthermore,  if  h  Is  small,  (4) 
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of  §5  be  replaced  bj  (2). 

If  one  step  of  tiM  Iteration  is  perfomed,  linear  interpolation 
method  can  now  be  used  to  continue  the  iteration  until  a  boundary 

a^^jXj  change! 

iign. 

In  thi!  wajt  performing  the  equivalent  of  one  hundred  or  more 
step!  at  a  time  the  hand  computer  oan»  in  the  early  stage!  of  the 
computation,  keep  pace  with  and  perhaps  even  gain  on  a  machine. 

When,  however,  the  values  are  close  to  an  actual  solution  the  sign 
changes  become  so  frequent  that  a  machine  is  more  efficient. 


is  reached,  that  is  to  say,  a  point  where  some  ^ 


§7.  A  7  X  7  Matrix. 

Let  us  consider  the  following  game  matrix 


which  has  the  unique  solution 


(2) 


Xi  ■  .111, 

X*  " 

.111, 

^9 

*5  -  *167, 

*6  ■ 

.056, 

*7 

.  l67 ,  X4  ■  .  058 , 

.333 


where  the  value  of  the  original  3x3  game  whleh  gave  rise  to  this 
.  skew  ■yntrie  matrix  la  1,  the  last  component* 

To  begin  with  we  take  h  •  *001  and  make  an  Initial  gueaa 

(3)  Xi  -  x«  -  •••  -  1/7 

< 

Following  we  present  the  first  two  pages  of  the  calculation  and 
the  subsequent  values  of  Xj. 
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1 


-^75 


I 


x(0) 

£ax 

x(l) 

£ax 

x(2) 

Zax 

x(5) 

£ax 

1 

.1429 

0 

1*25 

-0010 

1419 

1*15 

^10 

2 

.1429 

5714 

1*55 

5700 

1^39 

1**3 

5632 

3 

.1429 

1429 

1*55 

1425 

1439 

1409 

1**5 

1393 

4 

.1429 

-4286 

1*25 

-4315 

1419 

-4327 

1*15 

-4339 

5 

.1429 

-4286 

1*25 

-4315 

1419 

-4327 

1*15 

-4339 

6 

.1429 

1429 

1*55 

1425 

1439 

1399 

0060 

1**5 

1393 

7 

.1429 

0 

1*25 

0030 

1429 

1*55 

0090 

x(90) 

£ax 

x(91) 

£ax 

x(95) 

Zxx 

x(96) 

£ax 

1 

.0926 

-0005 

0922 

-0005 

0907 

-0005 

0905 

-0005 

2 

.1808 

2827 

1811 

2804 

1822 

2718 

1828 

2702 

3 

.1808 

0049 

1811 

0038 

1822 

-0003 

1818 

-0013 

4 

.0926 

-5^29 

0922 

-5438 

0907 

-5471 

0905 

-5489. 

5 

.0926 

-5^29 

0922 

-5438 

0907 

-5471 

0905 

-5449 

6 

.1808 

0049 

1811 

0038 

1822 

-0003 

1818 

-0013 

7 

.1803 

2646 

1806 

2667 

1817 

2745 

1823 

2749 

x(265) 

Zax 

x(266) 

Sax 

x(278) 

Sax 

x(279) 

Sax 

1 

.0601 

-0005 

0600 

0586 

-0005 

0586 

-0005 

2 

.2893 

0117 

2897 

0108 

2947 

-0012 

29^4 

-0019 

3 

.1203 

-1686 

1201 

1172 

-1770 

1171 

-1777 

4 

.0601 

-8684 

0600 

0586 

-8846 

0586 

-8827 

5 

.0601 

-1924 

0600 

0586 

-1746 

0586 

-1735 

6 

.1203 

-1686 

1201 

1172 

-1770 

1171 

-1777 

7 

.2888 

5*96 

2892 

29^2 

3533 

2949 

3528 

*(529) 

Sax 

x(330) 

Seuc 

x(331) 

Sax 

x(431) 

Sax 

1 

..0556 

0500 

0565 

0509 

0574 

0518 

1454 

1418 

2 

.2799 

-0519 

2793 

-0517 

2787 

-0515 

2227 

-0415 

3 

.1111 

-2187 

1109 

-2172 

1107 

-2157 

0887 

-0737 

4 

.0556 

-7897 

0555 

-7870 

0554 

-7843 

0444 

-52<3 

5 

.0556 

-1145 

0555 

-1134 

0554 

-1123 

0444 

0097 

6 

.1111 

-2187 

1109 

-2212 

1107 

-2237 

0887 

-4777 

7 

.5299 

3353 

3302 

3336 

3305 

3319 

3645 

1659 

CM  lAVO  r-t  CM  ITWO 


•H  M  invo  r«-  fH  CM  ^TWO  1^ 


x(432)  Sax 

1  .1460  1424 

2  .2220  -0412 

3  .0804  -0714 

4  .0443  -5236 

5  .0453  0108 

6  .0884  -3302 

7  .3644  1632 


x(433) 

Sax 

x(464) 

1466 

1430 

1640 

2213 

-0409 

2008 

0881 

-0691 

0800 

0442 

-5209 

0402 

0462 

0119 

0729 

0881 

-4747 

0800 

3643 

1603 

3612 

Sax 

x(463) 

Sax 

1643 

1609 

2002 

-0362 

-0003 

0798 

0013 

0401  -4397 
0737  0419 
0798  -4209 

3611  0815 


x(466)  Sax  x(467)  Sax  x(333)  Sax  x(334)  Sax  x(336)  Sax 


.1648  1393 

1631 

1875 

. 1994  -0383 

19^ 

0812 

1458 

.0803  0034 

1254 

. 0399  -4339 

.0744  0417 

0397 

0291 

0751 

0415 

1213 

.0793  -4272 

0792 

0581 

.3607  0803 

3603 

•0791 

3326 

0522  187Q  0300  1887  0439 

-^87  1434  -0277  1443  -0237 

1346  1260  1371  1272  1421 

-1833  0290  -1819  0288  -1788 

0329  1219  0208  1231  0296 

-3146  0379  -3127  0376  -3089 

-0086  3316  -0095  3296  -0113 


x(349)  Sax 
.1943  0179 

.1389  -0119 
.1353  1737 

.0276  -1391 

. 1313  0131 

.0354  -2867 

.3166  -0236 


x(559)  Sax 

1984  -0026 
1347  -0013 
1412  2002 
0268  -1442 
1373  0048 
0537  -2632 
3071  -0329 


x(360)  Sax 

1980 

1344  -0017 

1419 

0267 

1380  0038 

0336 

3063 


x(361)  Sax 
1976 

IIU 

0266 

1387  0028 

0335 

3039 


x(362)  Sax 
1972 

1338 

1433 

0263 

1394  0018 

0334 

3033 


x(563) 

Sax 

x(564) 

Sax 

x(565) 

Sax 

1 

.1968 

1964 

-0121 

1962 

-0143 

2 

.1335 

1332 

-0025 

1^3^ 

-0024 

3 

.1440 

1447 

2032 

2090 

4 

.0264 

0263 

-1372 

0263 

-1361 

3 

.1401 

0008 

1408 

-0002 

1407 

-0011 

6 

.0533 

.3047 

0532 

-2482 

0531 

-2298 

7 

3041 

-0324 

3038 

-0314 

-12- 
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§8.  A  21  z  21  Matrix* 

With  the  aiii  of  testing  the  convergence  of  the  value,  we 
obtained  the  following  21  x  21  matrix  .with  value  *3  frow  Ruth  Vagner* 


11 

12 

13 

14 

15  16 

17 

GO 

19  20  21 

1 

2 

3 

2 

2 

2 

1 

3 

3 

3 

2  -1 

2 

1 

2 

2 

2 

3 

2 

1 

2 

2 

3  -1 

3 

2 

2 

2 

3 

2 

2 

3 

2 

3 

2  -1 

4 

2 

2 

1 

2 

2 

2 

1 

2 

2 

3  -1 

5 

o 

2 

1 

2 

2 

2 

3 

5 

2 

1 

2  -1 

6 

3 

2 

2 

2 

1 

2 

1 

2 

2 

2  -1 

7 

1 

3 

1 

3 

1 

3 

2 

5 

1  -1 

8 

1 

2 

2 

2 

2 

2 

1 

2 

2 

2  -1 

9 

1 

2 

1 

2 

3 

2 

1 

2 

2 

3  -1 

10 

2 

1 

2 

1 

2 

2 

3 

2 

1 

2  -1 

11 

-2  -1 

-2-2-2 

-3  -1  -1 

-1 

-2 

1 

12 

-3  -2 

-2-2-1 

-2-3-2 

-2 

-1 

1 

13 

-2  -2 

-2-1-2 

-2-1-2 

-1 

-2 

1 

14 

-2  -2 

-3-2-2 

-2-3-2 

-2 

-1 

1 

15 

-2  -3  -2  -2  -2 

-1  -1  -2 

-3  -2 

1 

16 

-1  -2 

-2-2-3 

-2-3-2 

-2 

-2 

0 

1 

17 

~3  -1 

-3  -1  -3 

-1  -2  -1 

-1 

-3 

1 

18 

-3 

-2-2-2 

-2-3-2 

-2 

t 

1 

19 

-3  -2 

-3  -2  -1 

-2-3-2 

-2 

-1 

1 

20 

-3 

-2-3-2 

-2-1-2 

-3  -2 

1 

21 

1  1 

1  1  1 

1  1  1 

1 

1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1 

-1  0 

123^56789  10 


following  are  the  results  of  the  ooHputations*  Observe  that  the 
of  git  to  the  neighborhood  of  *3  is  extrewelj  rapid*  j 
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